Video frame processing method

ABSTRACT

A video frame processing method, which comprises: (a) capturing at least one first video frame via a first camera; (b) capturing at least one second video frame via a second camera; and (c) adjusting one candidate second video frame of the second video frames based on one of the first video frame to generate a target single view video frame.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the provisional application, which is U.S.Provisional Application No. 61/803,881, filed 2013 Mar. 21, and isincluded herein by reference.

BACKGROUND

Conventionally, a camera captures a frame via a camera parameter. Thecamera parameter can have, for example, a frame capturing parameter suchas an exposure time or a frame rate. Exposure time (or named shutterspeed) is the effective length of time a camera's shutter is open.Exposure time along with the aperture of the lens (also called f-number)determines the amount of light that reaches the film or an image sensorinside the camera. Long exposure time will cause image blur easily. Onthe contrary, short exposure time will cause image dark or noise easily.The aperture or camera sensor is always small in a camera phone (or asmartphone). When captured video frame resolution increases, the amountof light of each pixel will decrease. Therefore, it is hard to set abalanced camera parameter to generate a fine video frame.

A stereo camera is a type of camera with two or more cameras with aseparate image sensor for each camera. A stereo camera is always used togenerate a multi-view video frame (ex. a 3D video frame) based on thevideo frames generated from different cameras. Also, different cameraparameters can be applied to different cameras. An electronic devicewith a stereo camera becomes more popular in recent years (ex. a smartphone with a stereo camera), since the user may hope he can capture astereo image at any time he wants.

SUMMARY

Therefore, one objective of the present application is to provide avideo frame processing method to generate a target single view videoframe from the video frames captured by a stereo camera.

One embodiment of the present application discloses a video frameprocessing method, which comprises: (a) capturing at least one firstvideo frame via a first camera; (b) capturing at least one second videoframe via a second camera; and (c) adjusting one candidate second videoframe of the second video frames based on one of the first video frameto generate a target single view video frame.

Another embodiment of the present application discloses: a video frameprocessing method, which comprises: (a) capturing at least one firstvideo frame via a first camera utilizing a first camera parameter; (b)capturing at least one second video frame via a second camera utilizinga second camera parameter, wherein the second camera parameter isdifferent from the first camera parameter; and (c) generating a targetsingle view video frame corresponding to a specific time point accordingto the at least one first video frame, the first camera parameter andthe second camera parameter.

In view of above-mentioned embodiments, the target single view videoframes can be generated from video frames with different cameraparameters. Therefore a better target single view video frame can beacquired.

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a video processing methodaccording to one embodiment of the present application.

FIG. 2 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 1, according to one embodiment of the presentapplication.

FIG. 3 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 1, according to another embodiment of thepresent application.

FIG. 4 is a schematic diagram illustrating a video processing methodaccording to one embodiment of the present application.

FIG. 5 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 4, according to one embodiment of the presentapplication.

FIG. 6 is a schematic diagram illustrating a video processing methodaccording to another embodiment of the present application.

FIG. 7 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 6, according to one embodiment of the presentapplication.

FIG. 8 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 6, according to another embodiment of thepresent application.

FIG. 9 is a schematic diagram illustrating a video processing methodaccording to another embodiment of the present application.

FIG. 10 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 9, according to one embodiment of the presentapplication.

FIG. 11 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 9, according to another embodiment of thepresent application.

FIG. 12 and FIG. 13 are schematic diagrams illustrating a videoprocessing method according to another embodiment of the presentapplication.

DETAILED DESCRIPTION

In the following embodiment, two cameras are taken for example toexplain the concept of the present application. However, please notemore than two cameras can be applied to this application.

FIG. 1 is a schematic diagram illustrating a video processing methodaccording to one embodiment of the present application. As shown in FIG.1, one or more first video frames FF_1, FF_2 . . . FF_5 are captured bya first camera, and one or more second video frames SF_1, SF_2 . . .SF_5 are captured by a second camera. In one embodiment, the firstcamera captures each first video frame from the perspective of the lefteye, the second camera captures each second video frame from theperspective of the right eye, but is not limited. For example, the firstcamera and the second camera may be two independent cameras compriseindependent buttons and batteries. However, the first camera and thesecond camera are not limited to physically independent cameras. In oneembodiment, the first camera and the second camera can be provided on asingle electronic device, for example, two lenses on a single electronicdevice such as a camera or a mobile phone, using a single image sensoror independent image sensors. The video processing method of the presentapplication is, for example, performed by the processing unit of thesingle electronic device.

The target single view video frames TSF_1, TSF_2 . . . TSF_5 are, forexample, the video frames to be output to a display for displaying, butnot limited. Such target single view video frame can be generated basedon only the first video frame, or both the first video frame and thesecond video frame, which will be described later.

FIG. 2 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 1, according to one embodiment of the presentapplication. As shown in FIG. 2, the video processing method comprises:

Step 201

Capture first video frames and second video frames by the first cameraand the second camera, respectively.

Step 203

Analyze the video quality for one of the first video frames and thevideo quality for one of the second video frames corresponding to theanalyzed first video frame. For example, analyze the video quality forthe first video frame FF_2 and the second video frame SF_2. A firstvideo frame and a second video frame which are corresponding to eachother are, for example, frames captured at the same or similar timepoint by the first and the second cameras, respectively. However, thepresent application is not limited thereto.

Many methods can be applied to analyze the video quality. For example,the blur level/the sharpness level of each entire video frame and/or theregion of interest (e.g., one or combination of the face region, thecenter region, and the auto-focus region) in each video frame can becomputed to analyze the video quality. Also, the video frame can becompared to a reference video frame to analyze the video quality.Besides, at least one of the following parameters of each entire videoframe and/or the region of interest can be computed to analyze the videoquality: noise, edge, dynamic range, blocking artifact, mean intensity,color temperature, scene composition, people face and/or animalpresence, image content that attracts more or less interest, andSpatial/temporal/Frequency masking.

Step 205

Determine if the video quality of the first video frame is better than aquality threshold. If yes, it means the video quality is good, go tostep 207, if not, it means the video quality is bad, go to step 209.

Step 207

Select the first video frame analyzed in the step 203 as the targetsingle view video frame. For example, the first video frame FF_1 isdetermined to have a video quality better than the quality threshold,therefore the first video frame FF_1 is selected as the target singleview video frame TSF_1.

Step 209

Determine if the video quality of the second video frame is better thanthe quality threshold. If yes, go to step 213, if not, go to step 211.

Step 211

Interpolate the target single view video frame from the first videoframe and the second video frame analyzed in the step 203. For example,both the first video frame FF_5 and the second video frame SF_5 aredetermined to have a video quality lower than the quality threshold,therefore the target single view video frame TSF_5 is interpolated fromthe first video frame FF_5 and the second video frame SF_5.

Step 213

Select the second video frame analyzed in the step 203 and warp it. Forexample, the first video frame FF_3 is determined to have a videoquality lower than the quality threshold and the second video frame SF_3is determined to have a video quality better than the quality threshold.Therefore, the second video frame SF_3 is warped based on the firstvideo frame FF_3 to generate a warped second video frame. To bespecific, the warp operation performed on the second video frame is toeliminate the difference between the second video frame and thecorresponding first video frame due to the different viewingperspectives.

For the convenience for understanding, the second video frame can benamed as a candidate second frame if the second video frame correspondsto the first video frame analyzed in the step 203. For example, if thefirst video frame FF_4 is analyzed in the step 203, the second videoframe SF_4 is named a candidate second video frame, and the first videoframe FF_4 is named a candidate first video frame.

Please note, in one embodiment, the step 203 only analyzes the firstvideo frame. In such embodiment, the steps 209, 211 can be removed, suchthat the step 205 can go to the step 213 if the result of the step 205is no.

Step 215

Select the adjustment result generated in the step 213 as the targetsingle view video frame. For example, in FIG. 1, the target single viewvideo frame TSF_3 is generated by warping the second video frame SF_3based on the first video frame FF_3.

The step 213 can be replaced by other steps. In one embodiment, thesecond video frame is warped, and the target single view video frame isgenerated via synthesizing the warped second video frame and the firstvideo frame corresponding to the warped second video frame. For example,in FIG. 1, the target single view video frame TSF_4 may be generated bywarping the second video frame SF_4 based on the first video frame FF_4firstly, and then synthesizing the warped second video frame and thefirst video frame FF_4. In another embodiment, the target single viewvideo frame is generated via interpolating the first video frame and thesecond video frame. For example, in FIG. 1, the target single view videoframe TSF_4 may be a new video frame which is interpolating from thefirst video frame FF_4 and the second video frame SF_4.

Alternatively, the steps 213, 215 can be replaced by: enhancing thefirst video image based on the second video frame to generate the targetsingle view video frame if the video quality of the first video frame isbad and the video quality of the second video frame is good. Forexample, if the first video frame FF_4 has a bad video quality and thesecond video frame SF_4 has a good video quality, the first video frameFF_4 is enhanced based on the second video frame SF_4 to generate thetarget single view video frame TSF_4.

Furthermore, the step 211 can be replaced by other steps. In oneembodiment, the first video frame analyzed in the step 203 is selectedas the target single view video frame, since both video qualities of thefirst and the second video frames analyzed in the step 203 are worsethan the quality threshold. Alternatively, one of the first and thesecond video frames analyzed in the step 203 having a better videoquality will be selected as the target single view video frame. That is,either the first video frame or the second video frame can be selectedas the target single view video if both the first video frame or thesecond video have video qualities worse than the quality threshold.

In the embodiment of FIG. 1, the frame rates of the first camera and thesecond camera are the same, and corresponding first video frames andsecond video frames are simultaneously generated (ex. FF_1/SF_1 aresimultaneously generated, and FF_2/SF_2 are simultaneously generated).However, it does not mean to limit the scope of the present application,the two cameras may have different frame rates and the correspondingfirst video frames and second video frames can be generated at differenttime points. Furthermore, the corresponding first video frames andsecond video frames may be generated at different time points even thetwo cameras have the same frame rates. For example, the capture of thefirst video frames and the second video frames is staggered in time, butwith an identical frame rate. Additionally, the time intervals betweenthe video frames can be different.

Additionally, in one embodiment the video quality analyzing anddetermining steps are removed from the flow chart in FIG. 2, as shown inFIG. 3. In such embodiment, one candidate second video frame (the secondvideo frame corresponding to the analyzed first video frame, asabove-mentioned description) is adjusted based on the first video frameto generate the target single view video frame. As shown in steps 313,315 in FIG. 3, the target single view video frame is the warpedcandidate second video frame. Alternatively, the target single viewvideo frame may be generated by synthesizing the warped second videoframe and the corresponding first video frame.

FIG. 4 is a schematic diagram illustrating a video processing methodaccording to one embodiment of the present application. As shown in FIG.4, the first video frames FF_1, FF_2, FF_3 and the second video framesSF_1, SF_2 are interleavingly captured by the first camera and thesecond camera, respectively. The first camera and the second camera are,for example, reside in a single electronic device. In one embodiment,the video frame rate for outputting the target single view video framesto the display is higher than which of the first camera and the secondcamera.

In the embodiment of FIG. 4, the target single view video frames can begenerated via following methods: select the first video frame or thesecond video frame as the target single view video frame withoutmodification (ex. FF_1 and TSF_1); warp or interpolate the second videoframe as the target single view video frame according to the first videoframe prior to or after the second video frame (ex. warp SF_1 accordingto FF_1 and FF_2); warp the first video frame and the second video frameto the same view to generate the target single view video frame (ex.SF_2 and FF_3 are warped to respectively generate TSF_4 and TSF_5); warpor interpolate the first video frame as the target single view videoframe according to the second video frame prior to or after the firstvideo frame, and warp or interpolate the second video frame as thetarget single view video frame according to the first video frame priorto or after the second video frame. Please note warp or interpolatedescribed in this application indicates at least one the warp operationand the interpolating operation is performed. Please note the videoquality analyzing and determining steps can be applied to the embodimentillustrated in FIG. 4, to determine which method for generating thetarget single vide video frame should be applied.

FIG. 5 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 4, according to one embodiment of the presentapplication. FIG. 5 comprises the following steps:

Step 501

Capture first video frames and second video frames by the first cameraand the second camera, respectively.

Step 503

Determine if the video frame captured at a specific time point by amajor camera exists. In this embodiment, the first camera is set as themajor camera. In another embodiment, however, the second camera is setas the major camera. If yes, go to step 505, if not, go to step 507.

Step 505

Select the video frame from the major camera as the target single viewvideo frame. For example, assuming that the specific time point is timepoints T_1, T_3, or T_5, the first video frames FF_1, FF_2 and FF_3exist, thus the first video frames FF_1, FF_2 and FF_3 are selected asthe target single view video frames TSF_1, TSF_3 and TSF_5, withoutmodification. In one embodiment, some modifications can be applied tothe first video frames FF_1, FF_2 and FF_3. For example, the process ofbrightness adjustment and/or sharpness enhancement can be performed tothe first video frames FF_1, FF_2 and FF_3, and then the modified firstvideo frames are selected as the target single view video frames.

Step 507

Select a video frame captured by a camera different from the majorcamera (e.g., the second camera). For example, assuming that thespecific time point is time points T_2 or T_4, the first video frames donot exist, thus the second video frames SF_1, SF_2 are selected.

Step 509

Adjust the video frame selected in the step 507 based on one or morevideo frames from the major camera captured prior to or after the videoframe selected in the step 507. For examples, assuming that one of thesecond video frames is selected in the step 507, the selected secondvideo frame is warped and/or interpolated according to one or more firstvideo frames prior to or after the selected second frame. Please notethe interpolate operation in this application is not limited to a videoframe directly adjacent to the selected video frame. For example, aninterpolated video frame corresponding to the time point T_4 may begenerated based on the second video frame SF_2 and at least one of thefirst video frames FF_2 and FF_3. Alternatively, the interpolated videoframe corresponding to the time point T_4 may be generated based on thesecond video frame SF_2 and the first video frame FF_1.

Step 511

Select the adjustment result generated in the step 509 as the targetsingle view video frame. In this embodiment, the second video framewhich is warped and/or interpolated is selected as the target singleview video frame.

In another embodiment, the step 509 in FIG. 5 can be removed, and thetarget single view video frame is the second video frame selected in thestep 507. For example, at the time points T_2 and T_4, the second videoframes SF_1 and SF_2 are selected to be the target single view videoframes TSF_2 and TSF_4, without modification. Alternatively, thebrightness, sharpness and/or other attributes of the second video framesSF_1 and SF_2 may be adjusted before being selected as the target singleview video frames TSF_2 and TSF_4.

In yet another embodiment, the step 505 in FIG. 5 can be replaced byother steps. For example, the first video frame corresponding to thespecific time point is warped and/or interpolated according to one ormore second video frames prior to or after the first video frame, thenselected as the target single view video frame for output.

In still another embodiment, all of the first video frames and thesecond video frames are warped to the same specific view, and thenselected as the target single view video frames for output. For example,in FIG. 4, the first video frames FF_1 to FF_3 and the second videoframes SF_1 and SF_2 are outputted interleavingly after being warped tothe same viewing direction.

In the embodiment of FIG. 4 the first camera and the second camera havethe same video frame rate, and the interval between the first videoframes or the second video frames are the same. However, the firstcamera and the second camera may have different video frame rates, andthe interval between the first video frames or the second video framesmay be different either.

The embodiments of FIG. 1 and FIG. 4 can further comprise: interpolatingat least two of the first video frames to generate a new first videoframe, and adjusts the candidate second video frame based the new firstvideo frame to generate the target single view video frame. Take thesecond video frame SF_2 in FIG. 1 for example, if no first video frameFF_2 in FIG. 1 exists or the quality of first video frame FF_2 is toobad, which is simultaneously captured, a new first video frame isgenerated based on the first video frames FF_1 and FF_3, and the secondvideo frame SF_2 is adjusted according to the new first video frame.

The above-mentioned embodiments can be summarized as: A video frameprocessing method, comprising: (a) capturing at least one first videoframe via a first camera; (b) capturing at least one second video framevia a second camera; (c) adjusting one candidate second video frame ofthe second video frames based on one of the first video frame togenerate a target single view video frame.

FIG. 6 is a schematic diagram illustrating a video processing methodaccording to another embodiment of the present application. In thefollowing embodiments, an A camera and a B camera are provided torespectively capture A video frames and B video frames. In FIG. 6, the Avideo frames are denoted as AF_1 to AF_3, and so on. And, the B videoframes are denoted as BF_1 to BF_5, and so on. The A video frames andthe B video frames are captured via different camera parameters. Forexample, the A camera has frame rate lower than which of the B camera,and the A video frames have video frame resolutions higher than which ofthe B video frames, as illustrated in FIG. 6. In one embodiment, the Avideo frames are in a Full HD format and captured at a frame rate of 15fps, while the B video frames are in a VGA format and captured at aframe rate of 30 fps. The video processing method can be applied togenerate and output the target single view video frames in the full HDformat at a frame rate of 30 fps. That is, a video frame resolution ofthe target single view video frame is the same as the first video frame,and a frame rate of the target single view video frame is the same asthe second video frame.

In one embodiment, one of the two cameras which can provide the highestvideo frame resolution will be selected as a major camera (e.g., the Acamera), and the target single view video frames can be generated viafollowing method: interleavingly selecting an A video frame withoutmodification and an interpolated B video image as the target single viewvideo frame. In FIG. 6, the interpolated B video images are denoted asBFM_1 to BFM_5, and so on. In this embodiment, the selection of the Avideo frame and the interpolated B video image is staggered in time.Each interpolated B video image is synthesized according to the B videoframe and at least one of the adjacent frames captured by the A cameraand/or the B camera. For example, the interpolated B video frame BFM_2is synthesized according to the second video frame BF_2 and at least oneof the A video frames AF_1 and AF_2, and the B video frames BF_1 andBF_3. In one embodiment, each interpolated B video frame beingsynthesized has the same video frame resolution as which of the A videoframe. By this way, the frame rate of the target single view video frameis larger than which of the first video frames, and the video frameresolution of the target single view video frame is larger than which ofthe second video frames.

FIG. 7 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 6, according to one embodiment of the presentapplication. The flow chart in FIG. 7 comprises the following steps:

Step 701

Capture A video frames and B video frames by the A camera and the Bcamera, respectively.

Step 703

Set the camera with a largest video frame resolution as the majorcamera. For example, in the embodiment of FIG. 6, the A camera is set asthe major camera.

Step 705

Determine if the video frame captured at a specific time point by themajor camera exists. If yes, go to step 707, if not, go to step 709.

Step 707

Select the video frame from the major camera as the target single viewvideo frame. For example, at time points T_1 and T_3, the A video framesAF_1 and AF_2 exist, thus the A video frames AF_1 and AF_2 are selectedas the target single view video frames. In one embodiment, the A videoframes AF_1 and AF_2 can be directly selected as the target single viewvideo frames. However, in another embodiment, some modification (ex.sharpness enhancement or brightness adjustment) can be performed to theA video frames first, and then the modified A video frames can beselected as the target single view video frames.

Step 709

Interpolate a new video frame from the video frame captured at thespecific time point by the camera different from the major camera and atleast one of the adjacent video frames captured by the A camera and/orthe B camera. It is assumed that the A camera is the major camera, thena new video frame is interpolated from the B video frame captured at thespecific time point and at least one of the adjacent video framescaptured by the A camera and/or the B camera, wherein the adjacent videoframes are the video frames captured at or near the specific time point.Taking the time point T_2 as an example, the adjacent video frames maybe one or a combination of the A video frames AF_1 and AF_2, and the Bvideo frames BF_1 and BF_3.

Step 711

Select the interpolated video frame as the target single view videoframe. For example, at the time point T_2, no A video frame exist, thusthe target single view video frame is generated from interpolating the Bvideo frame BF_2, and at least one of B video frames BF_1, BF_3, and Avideo frames AF_1, AF_2.

In another embodiment, one of the two cameras which can support thehighest video frame rate will be selected as a major camera (e.g., the Bcamera), and the target single view video frames in FIG. 6 can befurther generated via following method: selecting each interpolated Bvideo frame as the target single view video frame. The interpolated Bvideo image is synthesized according to the corresponding B video frameand at least one of the A video frames captured at or near the time atwhich the B video frame being captured. For example, the interpolated Bvideo frame BFM_1 is synthesized according to the B video frame BF_1 andthe A video frames AF_1. In another example, the interpolated B videoframe BFM_2 is synthesized according to the B video frame BF_2 and atleast one of the A video frames AF_1, AF_2. In another embodiment, theinterpolated B video image is generated according to the corresponding Bvideo frame and at least one adjacent frames captured by the A cameraand/or the B camera. By this way, the video frame rate of the targetsingle view video frame is the same as which of the A video frames, andthe video frame resolution of the target single view video frame islarger than which of the B video frames.

FIG. 8 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 6, according to another embodiment of thepresent application. The flow chart in FIG. 8 comprises the followingsteps:

Step 801

Capture A video frames and B video frames by the A camera and the Bcamera, respectively.

Step 803

Set the camera with a largest frame rate as the major camera. Forexample, in the embodiment of FIG. 6, the B camera is set as the majorcamera.

Step 805

For each video frame captured by the major camera, interpolate a newvideo frame from the video frame captured by the major camera and atleast one of the corresponding video frames captured by the cameradifferent from the major camera and the adjacent video frames capturedby the major camera. It is assumed that the B camera is the majorcamera, then for each B video frame, interpolate a new video frame fromthe B video frame and at least one of: the corresponding A video framesand the adjacent B video frames. In which, the corresponding A videoframes are video frames captured at or near (e.g., prior to or after) atime point that the B video frame is captured, and the adjacent B videoframes are video frames captured prior to or after the B video frame.Taking the B video frame BF_3 in FIG. 6 as an example, the correspondingA video frames may be the A video frame AF_2, and the adjacent B videoframes may be the B video frames BF_2 and BF_4. The new video frame(i.e., the interpolated B video frame BFM_3) is interpolated from the Bvideo frame BF_3 and at least one of the A video frame AF_2 and the Bvideo frames BF_2 and BF_4.

The video frame resolution of the interpolated video frame is largerthan which of the B video frame. For example, each of the interpolated Bvideo frames BFM_1 to BFM_5 has video frame resolution the same as whichof the A video frames AF_1 to AF_3, as shown in FIG. 6.

Step 807

Select the interpolated video frame as the target single view videoframe. For example, in FIG. 6, the interpolated B video frames BFM_1 toBFM_5 are selected as the target single view video frames TSF_1 toTSF_5.

FIG. 9 is a schematic diagram illustrating a video processing methodaccording to another embodiment of the present application. Theembodiment of FIG. 9 is similar with the embodiment of FIG. 6, onedifferent between the embodiments of FIG. 6 and FIG. 9 is that the Avideo frames and B video frames have different frame capturingparameters in FIG. 9 rather the video frame resolutions in FIG. 6. Theframe capturing parameters can comprises at least one of: capturingtime, exposure time, depth of field, focus, ISO speed, and whitebalance.

FIG. 10 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 9, according to one embodiment of the presentapplication. The flow chart in FIG. 10 comprises:

Step 1001

Capture A video frames and B video frames by the A camera and the Bcamera, respectively.

Step 1003

Set the camera with a lowest video frame rate as the major camera. Forexample, in the embodiment of FIG. 9, the A camera is set as the majorcamera.

Step 1005

Determine if the video frame captured at a specific time point by themajor camera exists. If yes, go to step 1007, if not, go to step 1009.

Step 1007

Select the video frame from the major camera as the target single viewvideo frame. For example, at time points T_1 and T_3, the A video framesAF_1 and AF_2 exist, thus the A video frames AF_1 and AF_2 are selectedas the target single view video frames, with or without modification asillustrated in the embodiment of FIG. 7.

Step 1009

Interpolate a new video frame from the video frame captured at thespecific time point by the camera different from the major camera and atleast one of the adjacent video frames captured by the A camera and/orthe B camera. It is assumed that the A camera is the major camera, thena new video frame is interpolated from the B video frame captured at thespecific time point and at least one of the adjacent video framescaptured by the A camera and/or the B camera, wherein the adjacent videoframes are the video frames captured at or near the specific time point.Taking the time point T_2 as an example, the adjacent video frames maybe one or a combination of the A video frames AF_1 and AF_2, and the Bvideo frames BF_1 and BF_3.

Step 1011

Select the interpolated video frame as the target single view videoframe. For example, at the time points T_2, no A video frame exist, thusthe target single view video frame is generated from interpolating the Bvideo frame BF_2, and at least one of B video frames BF_1, BF_3, A videoframes AF_1, AF_2.

FIG. 11 is a flow chart illustrating the steps for the video processingmethod illustrated in FIG. 9, according to another embodiment of thepresent application. The flow chart in FIG. 11 comprises the followingsteps:

Step 1101

Capture A video frames and B video frames by the A camera and the Bcamera, respectively.

Step 1103

Set the camera with a largest frame rate as the major camera. Forexample, in the embodiment of FIG. 9, the B camera is set as the majorcamera.

Step 1105

For each video frame captured by the major camera, interpolate a newvideo frame from the video frame captured by the major camera and atleast one of the corresponding video frames captured by the cameradifferent from the major camera and the adjacent video frames capturedby the major camera. It is assumed that the B camera is the majorcamera, then for each B video frame, interpolate a new video frame fromthe B video frame and at least one of: the corresponding A video framesand the adjacent B video frames. In which, the corresponding A videoframes are video frames captured at or near (e.g., prior to or after) atime point that the B video frame is captured, and the adjacent B videoframes are video frames captured prior to or after the B video frame.Taking the B video frame BF_3 in FIG. 9 as an example, the correspondingA video frames may be the A video frame AF_2, and the adjacent B videoframes may be the second video frames BF_2 and BF_4. The new video frameis interpolated from the B video frame BF_3 and at least one of the Avideo frame AF_2 and the B video frames BF_2 and BF_4.

Step 1107

Select the interpolated video frame as the target single view videoframe.

In this embodiment, the video frame rate of the target single view videoframes is the same as the B video frames. Furthermore, the video qualitycan be enhanced by synthesizing the B video frames with the A videoframes. For example, when the brightness for some dark regions of the Bvideo frame is too low, it may be compensated based on a correspondingregion of the A video frame.

Please note the A video frames and the B video frames of the embodimentsin FIG. 6 and FIG. 9 can be interleavingly captured, as the embodimentof FIG. 4. In this embodiment, the video frame rate of the first camerais identical to the video frame rate of the second camera.

It would be appreciated that the brightness of the video frame isaffected by factors such as the exposure time and/or the ISO speed. Inanother embodiment of the present application, the camera parameter ofthe A camera and the camera parameter of the B camera are the exposuretime and/or the ISO speed, and are set differently. In this embodiment,assuming that the A camera is selected as the major camera, then the Avideo frame can be compensated by the corresponding B video frame. To bespecific, for the A video frame corresponding to each specific timepoint, at least one region to be enhanced in the A video frame isdetermined according to the camera parameter of the A camera, and atleast one region to be referenced in the B video frame corresponding tothe specific time point is determined according to the camera parameterof the B camera. Accordingly, the at least one region to be enhanced inthe A video frame can be enhanced based on the at least one region to bereferenced in the B video frame.

For example, assuming that some region in the A video image isoverexposed while the brightness of the other region is fine due to thecamera parameter of the A camera, and some region in the B video imageis over dark while the brightness of the other region is fine due to thecamera parameter of the B camera. The overexposed region (i.e., theregion to be enhanced) in the A video image can be enhanced by thebright parts (i.e., the region to be referenced) in the corresponding Bvideo image.

For another example, assuming that some region in the A video image isover dark while the brightness of the other region is fine due to thecamera parameter of the A camera, and some region in the B video imageis overexposed while the brightness of the other region is fine due tothe camera parameter of the B camera. The over-dark region (i.e., theregion to be enhanced) in the A video image can be enhanced by the darkparts (i.e., the region to be referenced) in the corresponding B videoimage.

FIG. 12 is another embodiment of the present application. In theembodiment of FIG. 12, the A video frames and the B video frames arecaptured via the same video frame rates, but the A video frames havebrightness higher than which of the B video frames. In such embodiment,the video frame(s) of one camera can be enhanced by the video frame(s)of the other camera. For example, assuming that the A camera is selectedas the major camera. If the A video frame AF_1 is too bright, it can beenhanced based on the B video frame BF_1 to generate a target singleview video frame TSF_1, since the B video frame BF_1 has lowerbrightness. In such case the B video frame BF_1 is regarded as theenhancing standard video frame, and the A video frame AF_1 is regardedas a video frame corresponding to the enhancing standard video frame. Onthe contrary, if the B video frame BF_2 is too dark, it can be enhancedand warped based on the A video frame AF_2 to generate a target singleview video frame TSF_2, since the A video frame AF_2 has higherbrightness. In such case the A video frame AF_2 is regarded as theenhancing standard video frame. Please note the enhancing here is notlimited to mean enhancing the whole video frame based on another wholevideo frame. The enhancing here can mean enhancing a region of interestfor one video frame based on a corresponding region of another videoframe. For example, if a people face in the A video frame AF_3 is toobright, it is enhanced by the people face in the B video frame. FIG. 13is an extended embodiment based on FIG. 12. In the embodiment of FIG.12, the brightness of A video frames is all higher than which of the Bvideo frames. However, in the embodiment of FIG. 13, the brightness ofsome A video frames is higher than which of corresponding B video frames(ex. A video frames AF_1, AF_4), but the brightness of some A videoframes is lower than which of corresponding B video frames (ex. A videoframes AF_2, AF_3). Therefore, for either the embodiment in FIG. 12 orthe embodiment in FIG. 13, brightness of the A video frame and the Bvideo frame is analyzed to determine if anyone has a predeterminedbrightness, and then the video frame having a brightness closer to thepredetermined brightness is determined as the enhancing standard videoframe, such that the other video frame is enhanced by the enhancingstandard video frame. The enhancing standard video frame can be appliedto all video frames of a video sequence after it is determined. However,if the enhancing standard video frame is captured by the major camera,the video frame enhanced by the enhancing standard video frame has to bewarped based on the corresponding video frame captured by the majorcamera, so as to generate the target single view video frame.

For example, in one embodiment of FIG. 12, if the A video frame isdetermined as the enhanced video frame, all A video frames are enhancedbased on video frames for B video frames. Alternatively, the enhancingstandard video frame can be applied to only some video frames for thevideo sequence after it is determined, and the enhancing standard videoframe can be changed in such case. For example, in FIG. 13, the A videoframes AF_2, AF_3 are enhancing standard video frame for the B videoframes BF_2, BF_3, but the B video frames BF_2, BF_3 are enhancingstandard video frame for the A video frames AF_2, AF_3. Such variationshould fall in the scope of the present application.

In view of the embodiments illustrated in FIG. 6-FIG. 11, the targetsingle view video frames can be generated by the video frames from themajor camera, based on the camera parameter such as video frame rate orvideo frame capturing parameter such as brightness or blur level.Therefore, the embodiments illustrated in FIG. 6-FIG. 11 can besummarized as: A video frame processing method, comprising: (a)capturing at least one first video frame via a first camera utilizing afirst camera parameter; (b) capturing at least one second video framevia a second camera utilizing a first camera parameter, wherein thesecond camera parameter is different from the first camera parameter;and (c) generating a target single view video frame corresponding toeach specific time point according to at least one first video frame,based on the first camera parameter and the second camera parameter.Please note the first camera and the second camera here only indicatetwo different cameras, and are not limited to have the same parameters(ex. resolution, frame rate . . . ) as which the first camera and thesecond camera.

Please note the corresponding relations of the first/second cameras andthe A/B cameras are not fixed. For example, in one embodiment the videoframe processing method comprises: determining if the first video framecorresponding to the specific time point exists; if yes, selecting thefirst video frame as the target single view video frame; and if not,interpolating a target single view video frame from one of the secondvideo frames corresponding to the specific time point and one or moreadjacent frames captured by the first camera and the second camera. Insuch case the first camera indicates the A camera, and the second cameraindicates the B camera.

In another embodiment, the video frame processing method comprises:generating the target single view frame according to one candidate firstframe, and at least one of: at least first video frame prior to or afterthe candidate first video frame, and at least second video frame priorto or after the candidate first video frame. In such case the firstcamera indicates the B camera, and the second camera indicates the Acamera.

In view of above-mentioned embodiments, the target single view videoframes can be generated from video frames with different cameraparameters and video frame capturing parameters. Therefore a bettertarget single view video frame can be acquired.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention. Accordingly, the abovedisclosure should be construed as limited only by the metes and boundsof the appended claims.

What is claimed is:
 1. A video frame processing method, comprising:capturing a first video frame via a first camera; capturing a secondvideo frame via a second camera; and generating a target video framethat is non-stereoscopic and corresponds to a viewing perspective of thefirst video frame, including: selecting the first video frame as thetarget video frame when image quality of the first video frame is betterthan a quality threshold; and generating the target video frame byadjusting the first video frame or the second video frame when the imagequality of the first video frame is not better than the qualitythreshold, including interpolating the first video frame and anothervideo frame captured via the first camera to generate a third videoframe, and adjusting the second video frame based the third video frameto generate the target video frame.
 2. The video frame processing methodof claim 1, further comprising: determining if the image quality of thefirst video frame is better than the quality threshold.
 3. The videoframe processing method of claim 1, wherein generating the target videoframe comprises: warping the second video frame based on the first videoframe.
 4. The video frame processing method of claim 1, wherein thefirst video frame and the second video frame are simultaneouslycaptured.
 5. The video frame processing method of claim 4, wherein thefirst camera and the second camera are configured operate at a samevideo frame rate.
 6. The video frame processing method of claim 1,wherein the first camera and the second camera interleavingly capturethe first video frame and the second video frame.
 7. The video frameprocessing method of claim 6, wherein a time point of the first videoframe is prior to or after a time point of the second video frame. 8.The video frame processing method of claim 6, wherein the first cameraand the second camera are configured operate at a same video frame rate.9. A video frame processing method, comprising: capturing first videoframes using a first camera; capturing second video frames using asecond camera; and for one of the first video frames, generating acorresponding target video frame that is non-stereoscopic andcorresponds to a viewing perspective of the one of the first videoframes, including: selecting the one of the first video frames as thecorresponding target video frame when image quality of the one of thefirst video frames is better than a quality threshold; and generatingthe corresponding target video frame when the image quality of the oneof the first video frames is not better than the quality threshold,including interpolating the one of the first video frames and anotherone of the first video frames to generate a third video frame, andadjusting one of the second video frames based the third video frame togenerate the corresponding target video frame.
 10. The video frameprocessing method of claim 9, wherein the one of the first video framesand the one of the second video frames are simultaneously captured. 11.The video frame processing method of claim 10, wherein the first videoframes and the second video frames have a same video frame rate.
 12. Thevideo frame processing method of claim 1, wherein the first video framesand the second video frames are interleavingly captured.
 13. The videoframe processing method of claim 12, wherein the first video frames andthe second video frames have a same video frame rate.